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Description 

PREDICTION SYSTEM BASED ON 
WEIGHTED EXPERT OPINIONS USING 
PRIOR SUCCESS MEASURES 

Background of Invention 

[0001] The present invention relates to prediction systems based 
on expert opinions. 

[0002] This invention enables the prediction of multiple events 
including, and not limited to, financial parameter perfor- 
mance (e.g. stock and bond prices, interest rates, currency 
exchange rates), sport events (e.g. scores, standings), 
natural phenomena (e.g. weather, resource level), and 
fuzzy events (e.g. speech and pattern recognition). In the 
context of this description, an expert is defined as any 
entity capable of putting forward an answer or opinion to 
the likelihood or quantitative value of an event. Therefore, 
experts would include human beings of varying degrees of 
skill and expertise as well as machines or devices capable 
of putting forward an answer or opinion. In the realm of 



machine applications, the use of independently trained 
classifiers and the aggregation of their outputs is com- 
monplace in technologies such as speech and pattern 
recognition. 

[0003] An event or the probability of an event occurring can be 
predicted according to different techniques, some more 
formal than others, ranging from deterministic simulation 
models to intuitive predictions. Standard formal prediction 
techniques include extrapolation, pattern analysis, and 
methods relying on the pooling of expert opinions. Pre- 
diction methods relying on expert opinions are common- 
place and ancient. Calling upon an oracle or the reading of 
tea leaves are what we could call informal approaches to 
predicting an event by relying on expert opinions. Among 
more rigorous approaches, the most prevalent method to 
elicit and aggregate the opinion of experts is the "Delphi 
Technique"developed by the RAND Corporation for the Air 
Force in the 1950's. According to this technique, a num- 
ber of experts are asked to submit anonymous opinions 
on a particular future event. As an example, an early fore- 
casting study using the Delphi Technique asked the ques- 
tion "In what year will the percentage of electric automo- 
biles among all automobiles in use reach 50%?"The Delphi 



Technique generally includes several rounds of inquiry 
with controlled feedback to the experts in order to reach 
consensus on a prediction. The prediction generated by 
the Delphi Technique is a statistically based group re- 
sponse. 

[0004] other methods for eliciting expert opinions as to an event 
occurring include conferencing methods. On-line confer- 
encing and workgroups have received particular attention 
recently. Exploratory modeling is another method, here a 
very large number of future scenarios is generated and 
evaluated by experts from a decision-making perspective. 

[0005] a fundamental problem with existing approaches to the 
prediction of events based on expert opinions is the need 
to ultimately make subjective calls that may hamper the 
end result. Foremost among these subjective calls, is the 
need to choose the experts from whom opinions are 
elicited. No matter how carefully experts are chosen, there 
is a chance of integrating experts whose opinions are un- 
reliable and, worse yet, there is the certainty of leaving 
out experts whose opinion may be crucial to the predictive 
process. 

[0006] Another flaw in existing approaches is the use of subjec- 
tive measures to pool the opinion of experts. Experts may 



disagree and our perception of the quality of the opinion 
of a particular expert may be wrong. The simplest ap- 
proach to aggregating or pooling opinions is simply to av- 
erage all opinions to calculate a predicted value or likeli- 
hood of an event. As mentioned, in the Delphi Technique 
several rounds of inquiry seek to coalesce the opinion of 
experts. This approach has been shown to work efficiently 
when the sources of error for different experts are inde- 
pendent. This constraint is clearly not met in the case of 
most experts, particularly humans. This may deviate the 
final prediction further from its true outcome, especially 
when far-from-average opinions are eliminated. The use 
of weights to integrate an expert's opinion to a body of 
personal knowledge is intuitive and is done on a daily ba- 
sis. Weighting expert opinions to calculate a final predic- 
tion has been used with methods such as the Delphi 
Technique. This approach raises, once again, the issue of 
making subjective calls in that weights may be assigned to 
expert opinions in an arbitrary manner often as part of the 

initial process of selecting the experts. 
Summary of Invention 

[0007] The primary object of this invention is to enable a system 
capable of predicting an event based on the pooling of 



expert opinions by using different quantitative measures 
of prior prediction performance to weigh each expert 
opinion. This invention not only allows for a better final 
prediction but also permits the integration of outside or 
less sought after experts. 

[0008] a second object of this invention is to enable a measure 
of an expert's performance, which may be used as a 
means of reward or disincentive. In the case of human ex- 
perts, the very basic urge to second-guess or "armchair 
coach" can not only be satisfied by this invention but also 
properly rewarded in the case of accurate predictions. 

[0009] Another object of this invention is to enable a system for 
decision making and of participatory rewards based on 
eliciting and polling the opinion of experts over the Inter- 
net or over any other extended computer network such as 
those internal to many organizations. 

[001 0] a further object of this invention is to enable apparatuses 
based on this method and capable of integrating different 
inputs weighted by prior performance when predicting the 
outcome of an event. 

[001 1] The first aspect of the present invention is a system com- 
prising a means to elicit expert opinions, a means to store 
prior expert opinions, a means to calculate measures of 



prior performance by the experts, a means to pool the 
opinions weighted by these measures of prior perfor- 
mance, and a means to issue predictions and expert per- 
formance measures. 

[0012] The second aspect of the present invention is a method 
comprising the steps of eliciting and receiving expert 
opinions, storing measures of prior performance, pooling 
expert opinions weighted by these prior performance 
measures according to an algorithm described below, is- 
suing predictions, tracking actual outcome of events, and 
updating the measure of performance for each expert. 

[0013] | n accordance with a preferred embodiment of this inven- 
tion, a prediction and reward system operates over a 
computer network by eliciting opinions from experts and 
issuing predictions and measures of expert performance. 
The expert opinions are weighted by a measure of prior 
performance and the weighted opinions are arithmetically 
averaged. In this preferred embodiment, the measure of 
prior expert performance is the average of inverse expo- 
nentials of absolute differences between prior expert 
opinion and actual event outcome. This average of inverse 
exponentials is a summation of inverse exponentials 
arithmetically factored by their time-series position in the 



sequence of previous opinions. Further the individual 
weights assigned to expert opinions are normalized by 
the same factor to make the sum of all weight equals to 1. 

[0014] A n alternate embodiment may be realized by using a dif- 
ferent pooling algorithm from the preferred arithmetic av- 
erage such as a weighted geometric average, weighted 
harmonic average, and other forms of weighted averages. 

[0015] | n a second alternate embodiment of this invention, the 
inverse exponential of absolute differences between ex- 
pert opinion and outcome is replaced by another mathe- 
matical function monotonously decreasing with the 
amount of absolute deviation between expert opinion and 
outcome. 

[0016] in another alternate embodiment, the measure of prior 
performance by the expert is first evaluated based on a 
test using historical data. 

[0017] a further alternate embodiment of this invention is a sys- 
tem where the first measure of an expert's performance is 
based on a mapping of prior prediction capabilities in a 
different realm of activity. For instance, an expert being 
utilized for the first time to offer predictions on future 
stock prices may have a starting weight factor based on 
her performance predicting bond prices. 



[0018] yet further alternate embodiments may be realized when 
operational analysis techniques or neural networks are 
used to determine the weights associated to the prior 
performance of experts. 

[0019] The above and other objects, features, and advantages of 
this invention will become apparent to, or may be learned 
by practice of the invention, by any person skilled in the 
art from this description in conjunction with the accompa- 
nying drawings in which preferred embodiments of the 
present invention are described and shown byway of il- 
lustrative examples. 
Brief Description of Drawings 

[0020] FIG. 1 is a block diagram of a prediction and reward sys- 
tem according to a preferred embodiment of the inven- 
tion. 

[0021] FIG. 2 is a flow diagram of a method for eliciting, weight- 
ing, and pooling expert opinions and for issuing predic- 
tions. 

[0022] FIG. 3 is a flow diagram of a method for eliciting, weight- 
ing, and pooling expert opinions and for issuing predic- 
tions and expert performance measures 

[0023] FIG. 4 displays the results of applying this invention's 

method as well as that of the prior art to the prediction of 



stock market performance 

[0024] FIG. 5 displays the accumulated error of this invention's 

method against that of the prior art for the example in 

FIG. 4 
Detailed Description 

[0025] Reference is now made to the drawings. Corresponding 
reference numbers are used throughout the figures to 
designate corresponding elements. 

[0026] The following is a description of illustrative embodiments 
and should not be construed in a limiting sense. Modifica- 
tions of the illustrative embodiments will be apparent to a 
person having ordinary skills in the art. 

[0027] FIG.l is a block diagram block of a prediction and reward 
system according to a preferred embodiment of the in- 
vention. The prediction and reward system comprises an 
input means 1 for experts to enter their opinions on an 
event outcome by responding to specific questions from 
input means 1. Register means 2 for capturing the data 
transmitted from input means 1. Storage means 3 for 
storing measures of prior performance by different ex- 
perts. Processing means 4 for processing the data regis- 
tered by register means 2 and the data stored in storage 
means 3. Processing means 4 weights the input opinion 



from experts stored in register means 2 by the corre- 
sponding measure of prior expert performance stored in 
storage means 3. Upon calculating a prediction based on 
the pooling of the weighted expert opinions, processing 
means 4 transmits the prediction to output means 5. In- 
put means 6 tracks the actual outcome of the predicted 
event and transmits the data to processing means 4. Pro- 
cessing means 4 compares the actual outcome of the 
event to the individual expert opinions, recalculates the 
expert performance weight, and transmits the recalculated 
weight to storage means 3 to be used as new measure of 
expert prior performance. Input/Output means 7 receives 
external inquiries on prior expert performance and upon 
validation by processing means 4 it issues measures of 
prior expert performance. Interconnect means 8 allow the 
transmission of data and other signals among all the 
components. 

[0028] input means 1 in this preferred embodiment is a website 
displaying specific questions and capable of validating the 
identity and capturing the opinion of an expert. Means 2, 
3, and 4 are respectively computer-based register, mem- 
ory, and processing means. Means 5, 6, and 7 are in the 
preferred embodiment email-based systems for the 



transmission of information to and from the prediction 
system. Further integration of the system may be achieved 
by making any or all of means 1 through 8 part of a single 
website. 

[0029] FIG. 2 is a flow diagram of a method for eliciting, weight- 
ing, and pooling expert opinions and for issuing predic- 
tions. The method comprises the following steps: Elicit 
and receive expert opinions. Store these expert opinions. 
Weigh the expert opinions by measures of prior perfor- 
mance. Pool the weighted opinions. Issue a prediction re- 
sulting from the pooling of weighted opinions. Update the 
measures of expert performance by comparing the expert 
prediction to the actual outcome of the event. 

[0030] FIG. 3 is a flow diagram of a method for eliciting, weight- 
ing, and pooling expert opinions and for issuing predic- 
tions and expert performance measures. In addition to the 
steps comprising the method described in FIG. 2, this 
method comprises the step of issuing a measure of an ex- 
pert's prior performance upon validating a request for 
performance measure. 

[0031] | n what follows an illustrative embodiment is described in 
detail. In this illustrative embodiment, a prediction system 
for the Down Jones Industrial Average of stock prices 



("DJI") is described and its performance compared to prior 
art techniques. The object of this system is to predict the 
value of the DJI one month in advance. For that purpose, 
ten experts are to submit their opinion on the value of the 
DJI not later than 21 trading days before a given date. In 
this example, the ten experts are (for illustration purposes 
non-intelligent experts are chosen), simple algorithms 
based on the analysis of historical data as shown in Table 
1. 

[0032] TABLE 1 



EXPERT NUMBER 


OPINION ALGORITHM FOR DJI 
VALUE IN 21 TRADING DAYS 


1 


Aggressive bull: Market up 5% 


2 


Aggressive bear: Market down 5% 


3 


Historical: Market up 1% over average for last 21 
days 


4 


Market trend for last 21 days will continue to hold 
true 


5 


Market trend for last 21 days will be reversed 


6 


Market trend for last quarter will continue to hold 
true 


7 


Market trend for last quarter will be reversed 


8 


Market trend for last year will continue to hold 
true 


9 


Market will stand at same value as one quarter 
ago 


10 


Market will stand at same value as one year ago 



[0033] The predicted DJI value in 21 trading days was first calcu- 
lated according to prior art technique; that is, the pre- 
dicted DJI value was calculated as the arithmetic average 
of the 10 expert opinions. Next, the predicted DJI value in 
21 trading days was calculated according to the method 
of this invention. 

[0034] First, for each day when expert number j had an opinion 
for the actual DJI value, a raw weight W was calculated as 
the inverse exponential of the absolute difference be- 
tween the expert opinion O and the actual DJI value D, as 

j 

represented by Formula 1. 

[0035] w = exp[-abs(0-D)] [Formula 1] 

j j 

[0036] Second, a normalizing factor F was calculated by adding 

all weights W., as shown in Formula 2. 
[0037] F = SUM[W :W ] [Formula 2] 

[0038] Third, normalized weights w were calculated by dividing 
the raw weights W. by the normalizing factor F, as in For- 
mula 3. 

[0039] w = W/F [Formula 3] 

j j 

[0040] Fourth, for expert number j a measure of prior perfor- 
mance m was calculated as the summation of inverse ex- 

j 

ponentials w. arithmetically factored by their time-series 



position in the sequence of previous ten opinions as cal- 
culated in Formula 4 where w is the latest weight for ex- 

ji 

pert j, w is the second to last weight and so on to w 

K J j2 a jlO 

which is the weight for expert j ten trading days earlier. 
[OO 41 ] m = (10*w +9*w +...+2*w +w )/55 [Formula 4] 

j jl j2 j9 jlO 

[0042] Formula 4 can be expressed in general terms as in For- 
mula 5 with N being 10 for the example above. 
[0043] m = {2/[(N+l)*N]}*SUM{[10*w ]:[(11-N)*w ]} [Formula 5] 

j *' jl jN 

[0044] Finally, expert j opinion O. is weighted by the measure of 
prior performance m and all weighted opinions are arith- 
metically averaged to calculate a predicted value P for the 
DJI as shown in Formula 6. 

[0045] P = sUM[(m *0 ):(m *0 )] [Formula 6] 

1 1 10 10 

[0046] Figure 4 displays the results of applying to the method of 
this invention as well as that of the prior art. The actual 
values of the DJI for the years 2002 and 2003 are shown 
as small circles linked by a solid line. The 21-trading-day 
predicted values according to the prior art are shown as 
open squares. The 21-trading-day predicted values ac- 
cording to this invention and calculated as the "P" values 
above are shown as diamonds linked by a short-dash line. 
It is clear that the results from this invention outperform 



those of the prior art. In fact, the prediction according to 
this invention is better than that according to the prior art 
78% of the time for the period considered. 
[0047] a further assessment of the advantage of this invention is 
shown in Figure 5 where the accumulated error of the 
prediction according to this invention is 47% of that ac- 
cording to the prior art. The accumulated error for this in- 
vention is less than that according to the prior art at all 
times. 

[0048] with regard to success measures, Table 2 summarizes the 
experts performance for the period considered. Notice 
that in the case of a non-weighted arithmetic average, the 
equivalent expert measure would be 0.1 for all experts. 
Here expert measures display a large range of values (0 to 
0.62) with some experts tending to dominate the overall 
prediction. Experts 6 ("Market trend for last quarter will 
continue to hold true"), 1 ("Aggressive bull: Market up 
5%"), and 3 ("Historical: Market up 1% over average for last 
21 days") show the largest average success measures. Ex- 
perts 2 ("Aggressive bear: Market down 5%") and 8 
("Market trend for last year will continue to hold true") 
show very low success measure. This quantitative infor- 
mation is an invaluable tool to evaluate expert perfor- 



mance. 

[0049] TABLE 2 



EXPERT 

A TT TH vTTl I'D 

NUMBER 


AVERAGE 
MEASURE 


ST. DEV. 
MEASURE 


MINIMUM 
MEASURE 


MAXI- 

MUM 

MEASURE 


EXPERT 
RANKING 


1 


0.18 


0.16 


0.01 


0.62 


2 


2 


0.01 


0.01 


0.00 


0.08 


10 


3 


0.18 


0.14 


0.00 


0.55 


3 


4 


0.16 


0.13 


0.00 


0.45 


4 


5 


0.08 


0.09 


0.00 


0.36 


7 


6 


0.19 


0.16 


0.00 


0.54 


1 


7 


0.10 


0.09 


0.00 


0.32 


6 


8 


0.02 


0.05 


0.00 


0.25 


9 


9 


0.06 


0.13 


0.00 


0.51 


8 


10 


0.13 


0.14 


0.00 


0.59 


5 



[0050] More predictable events than stock market performance 
and the use of intelligent experts allow for heightened 
performance by this prediction system and method. 



